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Abstract 

In this article we share how a district-level technology integration specialist used literature on 
implementing student response systems (SRS) for formative assessment, based on Desimone's 
(2009) core features of professional development design, Guskey's Levels of Professional 
Development Evaluation (1998, 2000, 2002), and Danielson's Observation Cycle (2007), to 
support 12 middle school teachers in using SRS in their classrooms. The work reported here 
provides an example of incorporating literature-based best practices to support teachers in 
effectively using technology in the classroom. The findings of this study indicate that the 
teachers learned to use the SRS technology and associated strategies to collect formative data 
and appropriately adjust instruction to meet learners' needs. This work has implications for SRS 
in K-12 classrooms, technology integration professional development, and for preservice 
teacher education. 

Keywords: Evaluation; Instructional design; Observation cycle; Professional development; 
Student response systems 


Introduction 

Student response systems (SRS), sometimes referred to as handheld clickers and recently designed 
as web-based or mobile apps, are educational technologies used to collect student data-diagnostic, 
formative, or summative. As textual and/or numerical data are collected via multiple choice or open- 
ended questions, SRS maintain anonymity among the student responders when teachers displays 
the charted data, but also provide individualized data for the instructor to review. Especially when 
literature-based strategies are used for SRS formative data collection, the learning process may be 
enhanced such that students are engaged and reflective while teachers receive information for 
modifying instructional pacing or content to immediately meet learners' needs (Polly, Rodgers, & 
Little, 2015; Penuel, Boscardin, Masyn, & Crawford, 2007). The ability for a technology to enable 
formative assessment in this manner is desirable to students and teachers alike. However, as with 
all educational technologies, teachers must be adequately prepared to use SRS or else they may go 
unused or, possibly, inappropriately used in classrooms (Williams & Kingham, 2003). 
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Purpose of the Study 


The impetus for the work reported herein was a district-wide purchase of SRS for use by teachers 
who historically had minimal opportunities for practicing technology integration because allotment 
of resources and support for teacher education at the district, state, and collegiate levels had not 
been consistent and, in some instances, had been non-existent. The teachers in this district had 
never used the SRS technology before yet administrators expected the technology to be used 
consistently and effectively. 

At the time of this work, the first author was a district-level technology integration specialist and a 
doctoral student. As part of her doctoral work she conducted a needs assessment of middle school 
teachers in the district as related to SRS and learned that they lacked knowledge on how to use the 
SRS for formative assessment and student engagement in their classrooms. She was aware that 
learning a new technology is difficult and effective integration would likely not occur without 
appropriate support (Williams & Kingham, 2003). Additionally, she knew that a lack of technology- 
supported pedagogical knowledge is an identified hindrance to technology integration and that 
conducting professional development is a strategy to overcome this barrier (Hew & Brush, 2007). 
Therefore, the overarching research question for the study was the following: In what ways will 
professional development based on literature-based best practices help middle school teachers to 
effectively implement SRS for formative assessment and student engagement? The two guiding 
research questions for this study evaluating the effect of the professional development are: 

1. In what ways are the teachers using the student response system for formative assessment? 

2. What effect did the teachers' instructional use of SRS have on student engagement? 

Consequently, she worked to carefully design and evaluate professional development for 12 middle 
school teachers to promote literature-based practices related to using SRS for formative 
assessment. In this article we share SRS literature used to inform the professional development as 
well as details about its design, evaluation, and outcomes. We then share the implications this work 
has for others in terms of SRS in K-12 classrooms, technology integration professional development, 
and preservice teacher education. 


Literature Review 


SRS for Formative Assessment 

Fies and Marshall (2006) assert that research is limited regarding the use of SRS for formative 
assessment. This is particularly true in primary and secondary schools, however, an exploratory 
study by Polly, et al. (2015) found that using SRS in an elementary mathematics classroom helped 
the teachers with formative assessment by providing data to inform the direction of instruction and 
track students' progress. For collecting these formative data, the literature suggests between three 
and six clicker questions for a 50-minute lesson (Beatty & Gerace, 2009; Bruff, 2009a). Pacing the 
questions appropriately between segments of content allows a teacher to create an opportunity for 
formative data collection in which students demonstrate their understanding of each segment. 
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Additionally, pacing the questions rather than asking them all at once keeps the students attentive 
and engaged throughout the lesson. 

There are also instructional strategies that, when used with SRS, will provide immediate formative 
assessment data. Each of these strategies incorporates SRS for formative assessment by providing 
an avenue to collect data and an approach for adjusting instruction. Three such strategies include: 

• Contingent/Agile Teaching (Bruff, 2009b; Beatty, Gerace, Leonard, & Dufresne, 2006; 
Draper & Brown, 2004) 

• Discussion Warm-up (Bruff, 2009a, 2009b) 

• Peer Instruction (Bruff, 2009a, 2009b; Mazur, 1997) 

Penuel et al. (2007) found that teachers tend to use SRS to gauge students' comprehension and 
determine misunderstandings, display response data for facilitating student reflection and 
discussion, and adjust instruction based on the response feedback -- all of which align with these 
three approaches and help promote student engagement. 


Contingent/Agile Teaching 

When using the Contingent/Agile Teaching strategy (Beatty et al., 2006; Bruff, 2009b; Draper & 
Brown, 2004), as the teacher instructs he or she collects real-time SRS data to monitor student 
achievement via the technology's charting feature and then uses that readily viewable data to 
modify instruction immediately. For example, teachers may proceed with a class discussion, show a 
focused video on the topic, or ask a follow-up clicker question to reteach or extend learning based 
on the students' responses. 

Such formative use of SRS is documented as effective in an empirical study by Draper and Brown 
(2004), in which collegiate students in a variety of disciplines identified this pedagogical approach 
as beneficial. According to these authors, the defining attribute of this strategy is the teacher's 
ability to vary instruction based on student needs rather than following a pre-determined 
instructional sequence. Therefore, as the teacher responds to students' learning needs by using a 
variety of approaches to help students master learning objectives, the students become engaged in 
the learning process. 


Discussion Warm-up 

The Discussion Warm-up strategy (Bruff, 2009a, 2009b) also helps a teacher monitor the students' 
progress. When using this strategy an initial question posed by the teacher stimulates the students' 
thinking about the topic thereby facilitating formative data collection. The teacher then gives the 
students sufficient time to submit his or her response to the question using the SRS, which helps to 
increase the participation of all learners as students have time to decide upon a response prior to 
the discussion. The increased participation that results facilitates student engagement as learners 
critically consider the content. Additionally, the resulting SRS data helps the teacher to identify the 
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students' understandings and misinterpretations for guiding the discussion appropriately, as well as 
to design future instruction to meet learners' needs. 


Peer Instruction 

The Peer Instruction strategy (Bruff, 2009a, 2009b; Mazur, 1997,) helps students learn content that 
is more difficult. For this strategy a teacher involves the students in an instructional sequence of 
questioning, peer interaction, and class-wide discussion to help the group understand the material. 
Once a question is posed, the anonymous response data are revealed to the class in the form of a 
chart. If there is a lack of understanding among the group, the teacher partners up the students for 
a peer discussion. After a few minutes of conversation, the students re-vote using the SRS. This 
interaction engages the entire class and provides formative data to the teacher, which helps guide 
the follow-up class discussion. An extensive survey study by Fagen, Crouch, and Mazur (2002) found 
significant learning gains and engagement among college students with use of this strategy. 


Web-based SRS 

While literature on using SRS for formative assessment in K-12 settings is scarce, the use of these 
tools for assessing student learning is likely to increase due to their availability on the Internet and 
their ability to engage learners during assessment. Web-based SRS tools are accessible to users from 
a phone, tablet, computer, or other device with Internet capability. A study on using the Nearpod 
(2017) application, a presentation tool with certain SRS capabilities, during elementary guided 
reading found that the technology helped teachers monitor their students' progress (Delacruz, 
2014). This particular technology allows teachers to ask multiple choice or open-ended questions, 
as well as have students draw a visual response to demonstrate their knowledge of a concept while 
collecting data instantaneously. In another study, survey data from students in a higher education 
setting suggested that the web-based SRS called Socrative (2017) provided instant feedback 
regarding students' knowledge (Walsh, 2014). The charted feedback from this technology not only 
helped students to self-reflect on their learning, but also helped teachers to formatively assess their 
students and adjust their instruction accordingly. Rebecca, Andrea, and Jermaine (2014) used the 
data collected via Socrative to immediately address their secondary students' misconceptions, 
which were evident from the responses received. Lastly, both Socrative and another SRS tool called 
Kahoot! (2017) have game-based features. A study by Wang (2015) conducted in a Bring Your Own 
Device (BYOD) classroom identified the competition element of Kahoot! as the main element for 
sustaining attention during instruction when using the tool. 

Although there are differences among SRS tools, one similarity among them is the capability for 
learners to respond to a question or a series of questions while the system collects and compiles 
their answers in real time. Therefore, regardless of whether it is a hand-held clicker or web-based 
software accessed from a computer or mobile device, SRS have the capability to provide formative 
assessment data to teachers that is instantaneous and helpful for guiding instruction, as well as 
engaging learners when strategies such as those mentioned herein are utilized. 
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Methods 


Participants 

The participants consisted of teachers from two middle schools in one southeastern school district 
who attended the professional development sessions and consented to participate in the study 
(N=12). The participants were 100% female; however, their years of experience teaching and college 
preparation level varied. The participants' mean years of experience teaching in Georgia at the time 
of the study was 16.08 with the frequency distribution as follows: 1 teacher with 1-5 years, 3 
teachers with 6-10 years, 4 teachers with 11-15 years, and 4 teachers with 21 or more years. The 
degree level of two participants was unknown (n=2), and all others held a graduate degree: masters 
(n=7), specialist (n=2), or doctorate (n=l). The participants' teaching assignments represented each 
grade level: 6 th grade (n=5), 7 th grade (n=3), 8 th grade (n=3), and one taught both 6 th and 7 th grade 
(n=l). The participants also represented every core content area - mathematics (n=3), science (n=3), 
social studies (n=3), and language arts (n=3). The "convenience sampling" strategy (Patton, 1987) 
was used, as the teachers were naturally present in the setting of the study. This sampling strategy 
is typical in evaluating professional development. 


SRS Professional Development Design 

Appropriately designed and implemented instruction for teachers can affect students' knowledge 
gains (Hill, 2007). An example is when professional development helps teachers use instructional 
technologies to enable best practices and thereby facilitate student success. The literature on 
professional development pinpoints specific valuable components for adult learners. A review of 
empirical studies by Desimone (2009) describes a core set of features for effective professional 
development in a variety of contexts. The framework components include (a) content focus, (b) 
active learning, (c) coherence, (d) duration, and (e) collective participation. Desimone (2009) 
suggests a relationship between increased teacher knowledge, instructional changes, student 
improvement, and these core professional development features. 

These core features were embedded in each professional development session conducted by the 
technology integration specialist, as shown in Table 1, and each session used a similar approach to 
provide consistency among the sessions. Additionally, the instruction was aligned to the state's 
teacher evaluation standards for effective formative assessment and technology integration. 
Therefore, during each session the first author facilitated discussions to help the teachers consider 
ways to use their SRS to collect formative data for assessing their students' progress. She also 
incorporated the three aforementioned literature-based strategies into the instruction to guide 
teachers' use of the SRS. The general instructional sequence of activities for the sessions was as 
follows: 

1. Introduce an aspect of the SRS technology and a literature-based strategy for using the 
tool formatively. 

2. Lead a discussion and cooperative activity to generate ideas regarding the impact of 
technology-integrated formative assessment on student achievement and engagement. 
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3. 


Facilitate an opportunity for teachers to experiment with the SRS technology and the 
literature-based assessment strategy. 

4. Provide time to work on a multimedia lesson that incorporates formative assessment 
strategies using SRS. 

This teacher-developed multimedia lesson was for their classroom use and demonstrated each 
teachers' ability to integrate formative assessment strategies via the SRS. The teachers worked on 
their multimedia lesson artifact during each of the sessions and received feedback from peers and 
the instructor during the process. Resources for helping the teachers incorporate SRS for formative 
assessment in their lesson designs were posted online and used during the sessions, as well as 
available for future reference. 

Table 1. Desimone's (2009) Core Features of Professional Development (PD) and SRS PD 


Core Features 

Definition 

Ways in Implemented in 
the Professional 
Development 

Other Supporting 

Literature for the Feature 

Content Focus 

PD programs should 
emphasize both subject 
matter content and how 

students learn the 

content. 

Teachers were provided 
suggested strategies for 
use in different content 
areas and practiced using 
SRS within their discipline. 

Desimone, Porter, Garet, 
Yoon, & Birman, 2002; 

Garet, Porter, Desimone, 
Birman, & Yoon, 2001; 

Hirsh & Killion, 2009; 

Hughes & Ooms, 2004; 
Sparks, 2002 

Active Learning 

PD programs should 
actively involve teachers 
in the learning process. 

Teachers developed and 
implemented a 
multimedia lesson 

artifact. 

Bradshaw, 2002; 

Desimone et al., 2002; 

Garet et al., 2001; Hirsh & 
Killion, 2009; Mouza, 2003 

Coherence 

What is taught in the PD 
program should align with 
state and district goals 
and standards for student 
learning. 

The PD was meaningful 
and explicitly aligned to 
using SRS to support state 
and district standards. In 
addition, the SRS were 
purchased district wide. 

Borko, 2004; Desimone et 
al., 2002; Garet et al., 

2001; Hill, 2007; Killion & 
Harrison, 2006; Knowles, 
Holton, & Swanson, 1998 

Duration 

PD programs of longer 
duration should be 
emphasized over short¬ 
term workshops. 

The PD included four 

sessions with 
opportunities for 
classroom 
implementation in 
between and culminated 
with an observation cycle 
that included support in 
the classroom during one 
or more lessons as 
requested by the teacher. 

Borko, 2004; Desimone et 
al., 2002; Garet et al., 

2001; Killion & Harrison, 
2006; Lee 2005 
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Collective 

Teachers should work 

Teachers had time during 

Ball, 1996; Desimone et 

Participation 

together during PD 

the professional 

al., 2002; Garet et al., 


programs. 

development sessions to 

2001; Guskey, 2003; Hill, 



collaborate on using the 
technology-based 
formative assessment 
strategies and were 
supported with feedback 
on their ideas. 

2009 


Data Collection: SRS Professional Development Evaluation Framework 

Guskey's Five Levels of Professional Development Evaluation (1998, 2000, 2002) was used as a 
framework for assessing the SRS professional development. This model includes five critical levels 
of information for evaluating professional development in educational settings. The data collection 
process becomes more complex at each level and generally requires success before the next level 
may be considered (Guskey, 2002). The evaluation begins with determining the participants' general 
reactions to the professional development, moves to gauging the learning experience, and then 
assesses how the organization supports change. The next level evaluates the teachers' application 
of new knowledge or skills, and the last level examines the impact on student learning, or in this 
case student engagement during the SRS lessons. When evaluating these last two levels, referred to 
as Levels 4 and 5, participants must be allowed enough time to incorporate the new ideas into their 
practice (Guskey, 2002). 

This study, part of a larger professional development evaluation study implementing all five 
evaluation levels, focuses on Level 4 and Level 5 of the evaluation model. Because data at these 
levels are collected after significant time has passed following the professional development, it is 
more difficult to implement these evaluation levels (Gordon, 1991). Consequently, many 
professional development evaluations focus only on the earlier levels (Cody & Guskey, 1997), and 
this study seeks to fill that gap in the literature. Additionally, these last two levels provide the most 
significant data regarding the effectiveness of the professional development being evaluated, as 
they address the participant's sustained implementation of their learning and how these new 
instructional practices resulting from the professional development ultimately impact students. 

Table 2 shows the guiding questions and data collection strategies aligned with each of the five 
levels of the larger evaluation study. This study focusing on Levels 4 and 5, bolded within the table, 
seeks to answer the overarching research question: In what ways will professional development 
based on literature-based best practices help middle school teachers to effectively implement SRS 
for formative assessment and student engagement? 


Observation Cycle for Data Collection at Levels 4 and 5 

Danielson's (2007) three-part observation cycle was used to determine how teachers used SRS for 
formative assessments (Level 4 question) and how teachers' use of these strategies influenced 
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student engagement (Level 5 question). The three-part cycle consisted of (1) a pre-observation 
interview, (2) a direct observation, and (3) a post-observation interview. 

Table 2. Alignment of the Five Evaluation Levels, Guiding Questions, and Data Collection Methods 
for the Larger Professional Development Evaluation 


Guskey's Five Levels of PD Guiding Questions Data Collection 

Evaluation 


Level 1: Participants' Reactions 


Level 2: Participants' Learning 


Level 3: Organization Support and 
Change 

Level 4: Participants' Use of New 
Knowledge and Skills 


Level 5: Student Learning 
Outcomes 


What are the teachers' 
perceptions about the 
professional development? 

In what ways did the teachers 
acquire the intended 
knowledge? 

In what ways does the 
organization help teachers 
implement the technologies? 

In what ways are the teachers 
using the student response 
system for formative 
assessment? 

What effect did the teachers' 
instructional use of SRS have on 
student engagement? 


Perceptions Questionnaire 


Teacher Multimedia Artifact 
Rubric 

Organization Support 
Questionnaire 

Observation Cycle: Pre- 
Observation Interview Protocol; 
Observation Field Notes Guide; 
Direct Observation Rubric 

Observation Cycle: Observation 
Field Notes Guide; Post- 
Observation Interview Protocol 


Pre-Observation Interview 

The pre-observation interview protocol was used to help the researcher understand the lesson that 
was about to be observed (Danielson, 2007). Standardized open-ended questions (Patton, 1987) 
were used to help the researcher gain insights into how and why the teacher designed the lesson 
and planned for the use of SRS within it. These questions enabled the researcher to triangulate the 
data gained through the observations (Fraenkel & Wallen, 2006). For example, one interview 
question asked teachers to explain how they planned to use SRS for formative assessment. The 
researcher was then able to observe the lessons and see how the teachers' intentions played out in 
practice. Pre-observation interviews occurred one to two days before the observations and lasted 
about 30 minutes. All of the interviews were transcribed by the first author. 


Direct Observation 

Direct observations occurred within a day or two after the pre-interview and to "get a full 
description and deep understanding" (Glense, 2006, p. 51), each observation lasted about an hour 
or the full length of the lesson. A rubric was designed to assess how well the teachers were meeting 
the two state teacher evaluation standards focused on in the professional development: 
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• The teacher uses accessible technology effectively to enhance student learning. 

• The teacher uses formative assessment strategies to monitor student progress and to adjust 
instruction in order to maximize student achievement. 

The rubric in Figure 1 includes descriptors for four levels of proficiency with each standard: Not 
Evident, Emerging, Proficient and Exemplary as well as indicators of what each level may look like in 
practice. The four different levels were assigned numerical codes from 0-3: Not Evident (0), 
Emerging (1), Proficient (2), Exemplary (3). 

Table 2. Direct Observation Rubric 


SBI. 1.5 The teacher uses accessible technology effectively to enhance student learning. 

AL 1.2 The teacher uses formative assessment strategies to monitor student progress and to adjust 
instruction in order to maximize student achievement on the state performance standards. 



SBI 1.5 

Not Evident 

Emerging 

Proficient 

Exemplary 

The teacher does not 

use accessible 
technology to 
enhance student 
learning. 

The teacher uses 

accessible 

technology; however, 
technology is used 
primarily with the 
whole class, select 
students, or as a tool 
for tutorials and drill. 

The teacher routinely 
uses accessible 
technology to 
enhance student 
learning and support 
their achievement. 

The teacher 
develops, 
implements, and 
evaluates a 
comprehensive 
approach for using 
accessible 
technology to 
enhance learning 
and achievement for 

all students. 

c 


The teacher does not 

The teacher uses 

The teacher 

The teacher 

0) 

E 


use formative 

some formative 

consistently uses 

consistently uses a 

CD 

> 


assessment 

assessment tasks and 

formative assessment 

variety of formative 

O 

Q. 


strategies either to 

tools to guide 

tasks and tools to 

assessment tasks 

£ 


monitor student 

adjustments of 

monitor student 

and tools to monitor 

M- 

o 


progress or to adjust 

whole-class 

progress over the 

student progress 

£ 

3 


instruction to meet 

instruction; however, 

course of most units 

over the course of 

3 

C 


student needs. 

formative assessment 

and to adjust 

all units and adjusts 

*4-» 

c 



is rarely used at the 

instruction to meet 

instruction to 

o 

u 



individual level or 

students' individual 

maximize student 

E 
a ) 



may be inconsistently 

learning needs 

achievement relative 

+-» 

i/> 

>* 



implemented. 

relative to GPS. 

to GPS for all 

to 

c 





learners. The 

o 

’■M 





teacher also involves 

(TJ 

_3 





students in decisions 

ru 

> 

LU 





about adjustments 

i_ 

CD 

rsj 




to instruction to 

_C 

u 

t— i 

i 




enhance their 

fU 

CD 

_ 

< 




learning. 
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Indicators: 

Indicators: 

Indicators: 

Indicators: 


The teacher... 

The teacher... 

The teacher... 

The teacher... 


o does not use the 

o employs clickers 

o delivers GPS 

o consistently 


provided 

for whole class 

aligned clicker 

implements 


technologies. 

summative 

questions during 

each of the 


o uses clicker 

assessment (e.g. 

direct 

proficient 


questions that 

giving a test) or 

instruction. 

indicators. 


are not guided 

game. 

o consistently uses 

o uses a variety of 


by GPS aligned 

o delivers clicker 

charts or other 

questioning 


lessons. 

questions during 

data to monitor 

strategies with 



direct 

individual 

clickers. 



instruction. 

student learning 

o uses clicker data 



o uses some charts 

and adjust 

to facilitate 



or other instant 

instruction. 

student 



feedback to 

o or uses the 

reflection about 



monitor student 

mobile 

their own 

<u £ 

u =! 


learning and 

interactive 

learning and 

-§ = 


adjust 

whiteboard 

involves them in 

> C 


instruction. 

screen to 

instructional 

m o 
o u 



monitor learning 

decisions. 

« 2 
aj 



of individual 


— -a 

g- 



students and the 


i .§ 

y _ 



class and adjust 


UJ < 



instruction. 



Note: Format modeled after Rubric for Evaluating North Carolina Teachers (North Carolina State Board of 
Education, 2007). Incorporates the 2011 state teacher evaluation standards used in the study. 


In addition to the rubric, a field notes guide (Bogdan & Biklen, 2007) was used to help the researcher 
further focus her observations. This guide included space for the researcher to document specifics 
of how the teachers used SRS for formative assessment and how engaged the students were in the 
lesson. 


Post-Observation Interview 

The post-observation interviews occurred immediately after or within a day of the observations, 
and standardized open-ended questions (Patton, 1987) enabled the researcher to understand the 
teachers' perspectives on how the lessons went and on student engagement within them. For 
example, teachers were asked to reflect on how they knew whether students were learning during 
the lesson and how they adjusted instruction accordingly. All of the interviews were transcribed by 
the first author. 


Data Analysis 

Thematic analysis, "a method for identifying, analyzing, and reporting patterns (themes) within 
data" (Braun & Clark, 2006, p. 6) was used to identify how teachers used SRS for formative 
assessments (Level 4 question) and how teachers' use of these strategies influenced student 
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engagement (Level 5 question). Braun and Clark's six-step process for using thematic analysis was 
employed, however, it is important to note that the process is iterative rather than linear (2006). 
First, all data (i.e. the pre- and post-interview transcripts, the observation rubric, and field notes) 
were read and reread to develop a strong familiarity with them. At this time the researcher also 
developed a frequency distribution of the participants' level of proficiency (i.e. Not Evident, 
Emerging, Proficient, and Exemplary) with the two state standards described above using the 
observation rubrics. Then, initial codes were generated keeping the focus of the research questions 
in mind. Themes were then generated from interpretative data analysis using an iterative process 
of identifying themes, searching for evidence to confirm (or disconfirm) them and defining, and 
naming and describing them. 


Results 

Results from Level 4 and Level 5 of the evaluation are presented here; however, it worth noting that 
within the context of the full evaluation all teachers reported positive perceptions about the 
professional development (Level 1), demonstrated proficiency in designing lessons aligned to SRS 
best practices (Level 2), and felt the district was providing adequate support for them to use SRS in 
their classroom (Level 3). 


Level 4: In What Ways Are the Teachers Using the Student Response System for Formative 
Assessment? 

Themes for Level 4 focused on how the teachers planned for and used SRS for formative assessment 
during instruction, including a variety of the PD strategies, and their demonstrated proficiency with 
the targeted teacher evaluation standards. 


Teachers Implemented a Variety of SRS Formative Assessment Strategies from the PD 

The data indicated that the majority of teachers became proficient in their ability to implement SRS 
for formative assessment to help students learn. During the pre-observation interview, coded 
transcription data revealed that the teachers were planning to use SRS for formative data collection 
and to adjust their instruction as a result. Most teachers implemented their plans accordingly during 
their lessons. 

During the lesson observations, teachers were implementing the literature-based questioning 
strategies with the SRS, as shown in Table 3, for the purpose of collecting data to use formatively 
during their lessons. Teachers were observed using the SRS charting data for monitoring student 
progress and making instructional adjustments as they incorporated the strategies from the 
professional development. 
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Table 3. Examples of Strategies Used During Observations 


Participant 

Content Area 

Lesson Topic 

SRS Formative Strategies 

1 

Language Arts 

Explanatory Writing 

None 

2 

Language Arts 

Literary Analysis 

Contingent/Agile Teaching 

3 

Language Arts 

Figurative Language 

Contingent/Agile Teaching 

4 

Mathematics 

Linear Equations 

Contingent/Agile Teaching 

5 

Mathematics 

Geometry 

Contingent/Agile Teaching 

6 

Mathematics 

Line Symmetry 

Contingent/Agile Teaching 

7 

Science 

Currents 

Discussion Warm-up 

8 

Science 

Botany 

Peer Instruction; Discussion Warm-up 

9 

Science 

Physical Science 

Discussion Warm-up 

10 

Social Studies 

Africa 

Contingent/Agile Teaching 

11 

Social Studies 

Cultures 

Discussion Warm-up 

12 

Social Studies 

Post-war Era 

Contingent/Agile Teaching 


As noted in Table 3, during the observation of Participant 6, the teacher implemented the 
Contingent/Agile Teaching clicker strategy (Bruff, 2009b; Beatty et al., 2006; Draper & Brown, 2004) 
during the instruction. The multimedia lesson had content slides alternating with embedded SRS 
questions, and the teacher used the resulting SRS charted data to identify and address weakness 
among the students, or opportunities for extension, before moving forward with the lesson. After 
the teacher presented a concept, she displayed a geometric shape or authentic image within the 
presentation. For each symmetry example, the students had a corresponding example on their 
paper for hands-on analysis. The teacher posed SRS questions to engage the students in reflecting 
on the lines of symmetry for each example and responded appropriately to the learners' needs 
based on the responses received via the SRS clickers. 

Another teacher, Participant 8, was observed using the Peer Instruction clicker strategy (Bruff, 
2009a, 2009b; Mazur, 1997) to monitor progress and assist students in learning botany concepts. 
As she discussed each of the flower's characteristics, she asked probing questions to determine the 
student's level of understanding of the functions, facilitated peer interaction by having students 
discuss their ideas with each other, and made connections to other scientific concepts through class¬ 
wide discussion. The teacher asked an appropriate number of SRS questions aligned to the content, 
and pacing was such that formative assessment data was readily available to help clarify 
misconceptions about the parts of the flower and their functions. 


Teachers Demonstrated Proficiency with the Targeted State Standards by Using SRS 

During the lesson observations, the data indicated that the majority of teachers were implementing 
research-based strategies found in the literature including collecting formative data, monitoring 
student progress, adjusting instruction, and engaging students in learning by involving them in the 
lesson. One way in which the majority of teachers demonstrated such strategies aligned to the 
teacher evaluation standards was by displaying the charted SRS data and discussing the results with 
the class. The teachers also used the SRS data to adjust the instruction through descriptive feedback 
by rephrasing concepts, explaining correct or incorrect answers, re-teaching difficult material, and 
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giving additional examples of misunderstood concepts. The teachers consistently asked additional 
questions beyond their planned SRS questions to facilitate discussion among the students. 

For example, Participant 6 was observed implementing SRS questions in a symmetry PowerPoint 
during a mathematics lesson. The presentation included examples, non-examples, and symmetry 
problems for students to solve including SRS questions directly aligned to the student performance 
standards. The students used the SRS to submit their answers to the questions about symmetry. 
The pacing of the questions and the use of technology was such that the teacher was able to 
immediately clarify misconceptions through re-teaching (Beatty & Gerace, 2009; Bruff, 2009b) and 
further questioning. 

Similarly, as Participant 8 implemented her botany lesson, she used the charting feature of the SRS 
to facilitate the students' reflection about their own learning and adjust instruction as needed. The 
teacher gave descriptive feedback by rephrasing definitions and concepts, explaining correct or 
incorrect answers, and giving additional examples of the science concepts. Additionally, this teacher 
included a SRS discussion question at the end of the botany lesson for gauging students' thoughts 
about the instructional strategies used in the lesson in order to involve them in future instructional 
decisions. 

Table 4 displays data for the participants' scores on the classroom observation rubric (Figure 1). The 
rubric's standards and example indicators were used to guide the scoring process. To earn a rating 
of Emerging, the SRS was used primarily for drill and practice with minimal use of data for formative 
assessment. A rating of Proficient was assigned when the teacher consistently used the SRS to ask 
questions aligned to the student performance standards, deliberately used the data to monitor 
learning and adjust instruction, and implemented one literature-based SRS strategy. An Exemplary 
rating indicated that the teacher used multiple strategies and also involved students in reflection on 
their learning based on the data reviewed in class. One teacher scored Emerging on the Direct 
Observation Rubric. Of the 12 teacher participants, 10 scored Proficient and one scored Exemplary 
on the rubric. These frequencies indicate that 8.33%, 83.33%, and 8.33% scored Emerging, 
Proficient, and Exemplary, respectively. 

Table 4. Frequency Data for the Observation Rubric 


Rating 

Frequency (N=12) 

Percent of Observed Population 

Not Evident 

0 

0.00% 

Emerging 

1 

8.33% 

Proficient 

10 

83.33% 

Exemplary 

1 

8.33% 


Level 5: What Effect Did the Teachers' Instructional Use Of SRS Have on Student Engagement? 

The themes for Level 5 address the participants' use of SRS for engaging their learners, as well as 
their thoughts about the experience in terms of the effectiveness of SRS for engagement during 
instruction. 
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Teachers were Proficient with Engaging Learners through Questioning Techniques 


The Level 5 observation field notes data indicated that the majority of teacher participants were 
proficient with engaging learners through questioning techniques. Examples of unengaging 
techniques include asking a question without follow-up or asking too many at once. These teachers 
asked an appropriate number of sufficiently-paced SRS questions (Beatty & Gerace, 2009; Bruff, 
2009a). Then, they followed with descriptive feedback such as rephrasing concepts, explaining 
correct or incorrect answers, and providing additional examples of the concept to aid learners' 
understanding. Their SRS questioning strategies also enabled engagement as students considered, 
responded, and discussed. 

Additionally, teachers engaged learners in reflection of their own learning by displaying the charted 
data and facilitating a discussion of it. The teachers asked questions such as, "Explain why letter A 
is not the correct option" or "Explain why you selected letter C." Such prompts by the teacher helped 
students to critically analyze their thinking about the content; and therefore, engaged them in the 
learning process. 


Student Behaviors Indicated Engagement during use of the SRS Technologies 

The students' observed behaviors indicated overall engagement during the learning process as they 
used the SRS. Coded student behaviors indicating engagement during use of the technologies 
included: Answering the teacher's questions with SRS technology, responding to the charted SRS 
data, and discussing the SRS questions with other students. The students made comments about 
the charts showing interest in the progress of the class and were on-task when discussing the 
content. 

Although the use of SRS produced an engaging learning experience, unengaged behavior was noted 
by the researcher. For example, occasionally one or more students were observed not using their 
SRS to respond to a teacher's question until prompted by the teacher. 


Teachers Believed that the Technology Facilitated Their Students' Engagement 

The post-observation interview data for Level 5 supported what was already observed in the 
classrooms regarding student engagement. During the post-observation interviews the teachers 
indicated that the technology helped the students to learn and reflect on their learning while 
providing instant feedback, both of which facilitated their students' engagement. The teachers 
affirmed that students were engaged with using the SRS due to the accountability for participation, 
even with the anonymity of the charting, as the students knew their teachers could review 
identifiable data afterwards. The teachers also indicated that their students remained engaged 
during the lesson as a result of using the literature-based formative assessment strategies with the 
SRS. 
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Specifically regarding learning outcomes resulting from student engagement with the SRS, the 
teachers stated that the SRS data helped reveal students' progress throughout the lesson, as well 
as to determine whether re-teaching the content was needed. The coded post-observation 
interview data also revealed that the teachers recognized their students were learning based on 
their SRS responses and engaging participation in follow-up discussions. 


Limitations 

The researcher attempted to increase the trustworthiness of the study and limit researcher bias by 
assessing interpretations of the data and remaining objective (Glesne, 2006). To increase validity 
the researcher collected data at two middle schools. Additionally, the researcher achieved validity 
through triangulation (Denzin, 2006) — carefully inspecting and interpreting multiple sources of 
data. 

Limitations include the sole focus on using SRS for formative assessment. This limited focus may 
have led to overlooking other valuable uses of the technology or additional instructional strategies. 
Another limitation of the study is the sample size because it may not accurately reflect the thoughts 
or abilities of all teachers in the two schools. Additionally, in determining the study's sample, the 
researcher failed to consider the teachers' ability, such as his or her previous difficulty with 
formative assessment strategies, limited use of the technology, or teaching expertise; however, the 
teachers had not received previous professional development on the topic. Because the participants 
volunteered for the study, their motivation or prior knowledge and instructional practice may vary 
from non-participants in the schools, which may have resulted in selection bias (Wayne, Yoon, Zhu, 
Cronen, & Garet, 2008). 


Discussion and Implications 

The work reported here is an example of how specialists on-site can integrate literature-based best 
practices for technology use and for professional development design and evaluation to support 
teachers in effectively using technology in the classroom. Implementing Guskey's research-based 
evaluation model provided insight into the effectiveness of the professional learning provided by 
the technology integration specialist and facilitated reflection on ways to improve future 
professional development in the district. The observation cycle helped gauge participants' use of 
their new knowledge and skills for effectively implementing the SRS for formative assessment such 
that future professional development could be designed to further their learning. This process also 
initiated consideration regarding the impact of such technologies on student engagement learning 
outcomes. As we will discuss here, the work has implications for SRS in K-12 classrooms, technology 
integration professional development, and for preservice teacher education. 


SRS in K-12 Classrooms 

Prior to this professional development, the middle school teachers reported mainly using SRS for 
quizzes, tests, homework collection, and for proposing question after question. While these 
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approaches may have merit, the literature suggests a more effective and engaging use of SRS 
involves implementing various strategies to formatively assess learning. The three SRS strategies 
taught during this professional development may help other teachers who desire to use the tool in 
ways that are more productive. Similarly, such techniques not only prove valuable for engaging 
learners in content, but also for helping students reflect on their own learning as well as helping 
each other learn. Given that differentiation strategies yield from formative data, using SRS to enable 
data collection and formative assessment provides teachers with information, in the form of 
individually identifiable response reports, to effectively differentiate instruction for their students. 
Since these tools are effective for formative assessment and student engagement, additional 
research on the use of these tools in K-12 settings could further support their use during instruction 
by identifying additional approaches for their use during the learning process. With the availability 
of web-based SRS, the previous expense of clicker hardware is eliminated, and conducting such 
research in BYOD or mobile device environments is becoming a viable alternative. 


Technology Integration Professional Development 

The literature-based strategies (Table 1) used in the design of this technology integration instruction 
are important to apply regardless of the tool being learned. More often than not, technology-based 
instruction focuses on step-by-step learning, which is generally perceived as ineffective. Whereas 
collaborative, active opportunities over an extended time for learning to use a particular technology 
for supporting learning can enable content mastery with a focus on standards. These strategies are 
helpful to teachers for sustaining their ability to implement a new technology during instructional 
practice. Additionally, an in depth professional learning evaluation helps to ensure this is the case, 
as teacher application of those new skills and the effect on student learning is determined. 

In particular, this work implicates the need for professional development on using SRS. Web-based 
SRS are commonly available and with appropriate support, teachers learn to use SRS in ways that 
benefit students. For that reason, it is imperative that we prepare teachers such that they are able 
to use these tools effectively in their classroom. In the survey study by Penuel, et al. (2007), 498 
elementary and secondary teachers reported that teachers who frequently used SRS in combination 
with various instructional strategies were more likely to have participated in professional 
development. These teachers were also more likely to report the SRS as effective tools in the 
classroom. A similar conclusion may be drawn about the educators discussed herein thereby 
supporting the need for preparing teachers to use SRS for formative assessment. Additional studies 
using Guskey's model to evaluate technology-based professional development in K-12 settings 
would add to the literature on effective professional development design and positively impact 
teachers' ability to innovatively use technology during instruction. 


Preservice Teacher Education 

Coursework for preservice teachers on integrating SRS can help these future educators to design 
lessons using the technology for assessing student learning and monitoring progress. To that end, 
preparing preservice teachers with formative assessment strategies for integrating SRS can facilitate 
their use of the technology to assess and monitor learning, which is an important skill for our 
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educators as evidenced by its inclusion in teacher effectiveness standards. The benefit of using SRS 
during this process is the timeliness of the data received and the tracking of individual progress, 
which cannot easily be accomplished through manual methods. Reasonably, knowing many 
preservice technology courses have been designed as tool-based courses, this study provides 
support for including pedagogical uses of technologies in such courses, as the instructional 
strategies incorporated into this study were most significant in facilitating the teachers' effective 
uses of the SRS. 

Modeling and teaching formative use of SRS in teacher education programs and courses helps 
candidates feel confident about using the tools upon entrance into the classroom. Additionally, 
instruction on using technology for formative assessment can support teacher candidates' 
completion of edTPA (Pearson, 2017), a performance-based assessment system for evaluating 
aspiring teachers, when specific pedagogical strategies are incorporated that help the preservice 
teachers assess student learning. Because the nature of the technology is to provide data to help 
teachers monitor individual and aggregate student progress thereby allowing the teacher to then 
modify instruction, using SRS may provide documented evidence of a teacher's ability to 
differentiate instruction based on formative data. Additionally, research on the impact of including 
SRS instruction in preservice teacher education courses could provide valuable information for 
teacher education programs. 


Conclusion 

The evaluation of professional development is essential. As noted by Guskey and Yoon (2009), 
critical evaluation of professional development helps determine its effectiveness, as studies reveal 
a connection between student learning and professional development adequately designed and 
implemented. Similarly, with technology-based learning it is critical that instructors remain 
knowledgeable of and incorporate the most recent tools available - in this case web-based SRS-to 
support effective teaching strategies. By incorporating SRS and other technologies into preservice 
programs and K-12 educator professional development, teachers will be prepared to use and will 
effectively incorporate educational technologies into their classrooms. 
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